25476: Variable stack now respects boundary of call opcodes, MAJOR

debug-visualizers/AmalgamInternalTypes.natvis

@@ -19,6 +19,7 @@
 			<Item Name="FREEABLE">(attributes &amp; 16) != 0</Item>
 			<Item Name="FREEABLE_TOP_NODE">(attributes &amp; 32) != 0</Item>
 			<Item Name="KNOWN_TO_BE_IN_USE">(attributes &amp; 64) != 0</Item>
+			<Item Name="SCOPE_BREAK">(attributes &amp; 128) != 0</Item>
 			<Item Name="attributes">attributes</Item>
 		</Expand>
 	</Type>

docs/basic_math.md

@@ -5,7 +5,7 @@ parent: Opcodes
 nav_order: 6
 ---
 
-### Opcode: `+` {#opcode-add}
+### Opcode: `+`
 #### Parameters
 `[number x1] [number x2] ...`
 #### Returns
@@ -31,7 +31,7 @@ Output:
 
 [Amalgam Opcodes](./opcodes.md)
 
-### Opcode: `-` {#opcode-subtract}
+### Opcode: `-`
 #### Parameters
 `[number x1] [number x2] ...`
 #### Returns
@@ -65,7 +65,7 @@ Output:
 
 [Amalgam Opcodes](./opcodes.md)
 
-### Opcode: `*` {#opcode-multiply}
+### Opcode: `*`
 #### Parameters
 `[number x1] [number x2] ...`
 #### Returns
@@ -91,7 +91,7 @@ Output:
 
 [Amalgam Opcodes](./opcodes.md)
 
-### Opcode: `/` {#opcode-divide}
+### Opcode: `/`
 #### Parameters
 `[number x1] [number x2] ...`
 #### Returns

docs/control_flow.md

@@ -141,7 +141,7 @@ Output:
 #### Returns
 `any`
 #### Description
-Evaluates `function` after pushing the `params` assoc onto the scope stack.  If `constraints` is specified and not false or null, it will constrain execution.  If `constraints` is true or an assoc, it will default all constraints to be on at reasonable values for small execution without access to any data beyond `params`.  They optional key-value combinations for `constraints` are as follows.  If "max_node_operations" is specified, it represents the number of operations that are allowed to be performed. If "max_node_operations" is 0, then an infinite of operations will be allotted, up to the limits of the current calling context.  If "max_node_allocations" is specified, it represents the maximum number of nodes that are allowed to be allocated, limiting the total memory, up to the current calling context's limit.   If "max_node_allocations" is 0 and the caller also has no limit, then there is no limit to the number of nodes to be allotted as long as the machine has sufficient memory.  Note that if "max_node_allocations" is specified while in a multithreaded environment, if the collective memory from all the executing threads exceeds the average memory specified by "max_node_allocations", that may trigger a memory limit for the call.  If "max_operation_depth" is 0 or infinite and the caller also has no limit, then there is no limit to the depth that opcodes can execute, otherwise "max_operation_depth" limits how deep nested opcodes will be called. If `return_warnings` is true (default is false), the result will be a tuple of the form [value, warnings, performance_constraint_violation], where warnings is a list of all warnings, and perf_constraint_violation is a string denoting the performance constraint exceeded (or .null if none)).  The keys "read_access" and "write_access" are boolean and control whether the execution can read from or write to entities and access their relevant permissions (e.g., to load files, make system calls).  If the parameter `return_warnings` is false, just the value will be returned.
+Evaluates `function` after pushing the `params` assoc onto the scope stack.  If `params` is missing or null, then it will allow access to the scope stack of the caller.  If `constraints` is specified and not false or null, it will constrain execution.  If `constraints` is true or an assoc, it will default all constraints to be on at reasonable values for small execution without access to any data beyond `params`.  They optional key-value combinations for `constraints` are as follows.  If "max_node_operations" is specified, it represents the number of operations that are allowed to be performed. If "max_node_operations" is 0, then an infinite of operations will be allotted, up to the limits of the current calling context.  If "max_node_allocations" is specified, it represents the maximum number of nodes that are allowed to be allocated, limiting the total memory, up to the current calling context's limit.   If "max_node_allocations" is 0 and the caller also has no limit, then there is no limit to the number of nodes to be allotted as long as the machine has sufficient memory.  Note that if "max_node_allocations" is specified while in a multithreaded environment, if the collective memory from all the executing threads exceeds the average memory specified by "max_node_allocations", that may trigger a memory limit for the call.  If "max_operation_depth" is 0 or infinite and the caller also has no limit, then there is no limit to the depth that opcodes can execute, otherwise "max_operation_depth" limits how deep nested opcodes will be called. If `return_warnings` is true (default is false), the result will be a tuple of the form [value, warnings, performance_constraint_violation], where warnings is a list of all warnings, and perf_constraint_violation is a string denoting the performance constraint exceeded (or .null if none)).  The keys "read_access" and "write_access" are boolean and control whether the execution can read from or write to entities and access their relevant permissions (e.g., to load files, make system calls).  If the parameter `return_warnings` is false, just the value will be returned.
 #### Details
  - Permissions required:  none
  - Allows concurrency: false

docs/logic_and_comparison.md

@@ -197,7 +197,7 @@ Output:
 
 [Amalgam Opcodes](./opcodes.md)
 
-### Opcode: `=` {#opcode-equal}
+### Opcode: `=`
 #### Parameters
 `[any node1] [any node2] ...`
 #### Returns
@@ -266,7 +266,7 @@ Output:
 
 [Amalgam Opcodes](./opcodes.md)
 
-### Opcode: `!=` {#opcode-not_equal}
+### Opcode: `!=`
 #### Parameters
 `[any node1] [any node2] ...`
 #### Returns
@@ -348,7 +348,7 @@ Output:
 
 [Amalgam Opcodes](./opcodes.md)
 
-### Opcode: `<` {#opcode-less_than}
+### Opcode: `<`
 #### Parameters
 `[number|string node1] [number|string node2] ...`
 #### Returns
@@ -398,7 +398,7 @@ Output:
 
 [Amalgam Opcodes](./opcodes.md)
 
-### Opcode: `<=` {#opcode-less_than_or_equal}
+### Opcode: `<=`
 #### Parameters
 `[number|string node1] [number|string node2] ...`
 #### Returns
@@ -464,7 +464,7 @@ Output:
 
 [Amalgam Opcodes](./opcodes.md)
 
-### Opcode: `>` {#opcode-greater_than}
+### Opcode: `>`
 #### Parameters
 `[number|string node1] [number|string node2] ...`
 #### Returns
@@ -514,7 +514,7 @@ Output:
 
 [Amalgam Opcodes](./opcodes.md)
 
-### Opcode: `>=` {#opcode-greater_than_or_equal}
+### Opcode: `>=`
 #### Parameters
 `[number|string node1] [number|string node2] ...`
 #### Returns
@@ -580,7 +580,7 @@ Output:
 
 [Amalgam Opcodes](./opcodes.md)
 
-### Opcode: `~` {#opcode-type_equal}
+### Opcode: `~`
 #### Parameters
 `[any node1] [any node2] ...`
 #### Returns
@@ -614,7 +614,7 @@ Output:
 
 [Amalgam Opcodes](./opcodes.md)
 
-### Opcode: `!~` {#opcode-type_not_equal}
+### Opcode: `!~`
 #### Parameters
 `[any node1] [any node2] ...`
 #### Returns

docs/opcodes.md

@@ -45,19 +45,19 @@
   - [or](./logic_and_comparison.md#opcode-or)
   - [xor](./logic_and_comparison.md#opcode-xor)
   - [not](./logic_and_comparison.md#opcode-not)
-  - [=](./logic_and_comparison.md#opcode-equal)
-  - [!=](./logic_and_comparison.md#opcode-not_equal)
-  - [<](./logic_and_comparison.md#opcode-less_than)
-  - [<=](./logic_and_comparison.md#opcode-less_than_or_equal)
-  - [>](./logic_and_comparison.md#opcode-greater_than)
-  - [>=](./logic_and_comparison.md#opcode-greater_than_or_equal)
-  - [~](./logic_and_comparison.md#opcode-type_equal)
-  - [!~](./logic_and_comparison.md#opcode-type_not_equal)
+  - [=](./logic_and_comparison.md#opcode-=)
+  - [!=](./logic_and_comparison.md#opcode-!=)
+  - [<](./logic_and_comparison.md#opcode-<)
+  - [<=](./logic_and_comparison.md#opcode-<=)
+  - [>](./logic_and_comparison.md#opcode->)
+  - [>=](./logic_and_comparison.md#opcode->=)
+  - [~](./logic_and_comparison.md#opcode-~)
+  - [!~](./logic_and_comparison.md#opcode-!~)
 ### Basic Math
-  - [+](./basic_math.md#opcode-add)
-  - [-](./basic_math.md#opcode-subtract)
-  - [\*](./basic_math.md#opcode-multiply)
-  - [/](./basic_math.md#opcode-divide)
+  - [+](./basic_math.md#opcode-+)
+  - [-](./basic_math.md#opcode--)
+  - [\*](./basic_math.md#opcode-*)
+  - [/](./basic_math.md#opcode-/)
   - [mod](./basic_math.md#opcode-mod)
   - [get_digits](./basic_math.md#opcode-get_digits)
   - [set_digits](./basic_math.md#opcode-set_digits)

docs/variable_definition_and_modification.md

@@ -718,8 +718,6 @@ Example:
 Output:
 ```amalgam
 [
-	{}
-	{x 1}
 	{a 1}
 ]
 ```

src/Amalgam/evaluablenode/EvaluableNode.h

@@ -66,8 +66,10 @@ class EvaluableNode
 		FREEABLE_TOP_NODE = 1 << 5,
 		//if true, then known to be in use with regard to garbage collection
 		KNOWN_TO_BE_IN_USE = 1 << 6,
-		ALL = HAS_EXTENDED_VALUE | NEED_CYCLE_CHECK | IDEMPOTENT | CONCURRENT
-				| FREEABLE | FREEABLE_TOP_NODE | KNOWN_TO_BE_IN_USE
+		//if true, then the scope stops at this node
+		SCOPE_BREAK = 1 << 7,
+		ALL = HAS_EXTENDED_VALUE | NEED_CYCLE_CHECK | IDEMPOTENT | CONCURRENT | FREEABLE | FREEABLE_TOP_NODE
+			| KNOWN_TO_BE_IN_USE | SCOPE_BREAK
 	};
 
 	//constructors
@@ -696,6 +698,18 @@ class EvaluableNode
 	}
 #endif
 
+	//if true, then the scope stops at this node
+	__forceinline bool IsScopeBreak()
+	{
+		return HasAttribute(Attribute::SCOPE_BREAK);
+	}
+
+	//sets whether the scope stops at this node
+	__forceinline void SetScopeBreak(bool scope_break)
+	{
+		SetAttribute(Attribute::SCOPE_BREAK, scope_break);
+	}
+
 	//returns true if value contains an extended type
 	__forceinline bool HasExtendedValue()
 	{

src/Amalgam/interpreter/Interpreter.cpp

@@ -79,7 +79,7 @@ EvaluableNodeReference Interpreter::ExecuteNode(EvaluableNode *en,
 	return retval;
 }
 
-void Interpreter::InterpretAndPushNewScopeStackNode(EvaluableNode *new_scope_node, bool interpret_with_new_scope)
+void Interpreter::InterpretAndPushNewScopeStackNode(EvaluableNode *new_scope_node, bool interpret_with_new_scope, bool scope_break)
 {
 	EvaluableNodeReference new_scope = EvaluableNodeReference::Null();
 	bool need_to_interpret_new_scope = false;
@@ -107,6 +107,7 @@ void Interpreter::InterpretAndPushNewScopeStackNode(EvaluableNode *new_scope_nod
 		//just in case a variable is added which needs cycle checks
 		new_scope->SetNeedCycleCheck(true);
 
+		new_scope->SetScopeBreak(scope_break);
 		scopeStack.push_back(new_scope);
 
 		return;
@@ -119,6 +120,7 @@ void Interpreter::InterpretAndPushNewScopeStackNode(EvaluableNode *new_scope_nod
 		//just in case a variable is added which needs cycle checks
 		new_scope->SetNeedCycleCheck(true);
 
+		new_scope->SetScopeBreak(scope_break);
 		scopeStack.push_back(new_scope);
 
 		auto new_scope_mcn = new_scope->GetMappedChildNodesViewOnAssoc();
@@ -159,6 +161,8 @@ void Interpreter::InterpretAndPushNewScopeStackNode(EvaluableNode *new_scope_nod
 		//just in case a variable is added which needs cycle checks
 		new_scope_on_stack->SetNeedCycleCheck(true);
 
+		new_scope_on_stack->SetScopeBreak(scope_break);
+
 		if(interpret_with_new_scope)
 			scopeStack.push_back(new_scope_on_stack);
 
@@ -301,11 +305,21 @@ EvaluableNode *Interpreter::GetScopeStackGivenDepth(size_t depth
 EvaluableNode *Interpreter::MakeCopyOfScopeStack()
 {
 	EvaluableNode stack_top_holder(ENT_LIST);
-	stack_top_holder.SetOrderedChildNodes(begin(scopeStack), end(scopeStack), true, false);
+
 	//set flags conservatively before copy
 	stack_top_holder.SetNeedCycleCheck(true);
 	stack_top_holder.SetIsIdempotent(false);
 
+	//find scope break and erase before that
+	size_t scope_start_index = scopeStack.size();
+	for(; scope_start_index > 0; scope_start_index--)
+	{
+		if(scopeStack[scope_start_index - 1]->IsScopeBreak())
+			break;
+	}
+	stack_top_holder.GetOrderedChildNodesReference()
+		= EvaluableNode::OrderedType(begin(scopeStack) + scope_start_index, end(scopeStack));
+
 	EvaluableNodeReference copied_stack = evaluableNodeManager->DeepAllocCopy(&stack_top_holder);
 
 #ifdef MULTITHREAD_SUPPORT
@@ -315,9 +329,20 @@ EvaluableNode *Interpreter::MakeCopyOfScopeStack()
 		auto &stack_nodes_ocn = copied_stack->GetOrderedChildNodesReference();
 		for(Interpreter *interp = callingInterpreter; interp != nullptr; interp = interp->callingInterpreter)
 		{
-			stack_nodes_ocn.insert(begin(stack_nodes_ocn), scopeStack.size(), nullptr);
-			for(size_t i = 0; i < scopeStack.size(); i++)
-				stack_nodes_ocn[i] = evaluableNodeManager->DeepAllocCopy(scopeStack[i]);
+			size_t parent_scope_start_index = interp->scopeStack.size();
+			for(; parent_scope_start_index > 0; parent_scope_start_index--)
+			{
+				if(interp->scopeStack[parent_scope_start_index - 1]->IsScopeBreak())
+					break;
+			}
+
+			stack_nodes_ocn.insert(begin(stack_nodes_ocn), interp->scopeStack.size() - parent_scope_start_index, nullptr);
+
+			for(size_t i = 0; i < interp->scopeStack.size() - parent_scope_start_index; i++)
+			{
+				stack_nodes_ocn[i]
+					= evaluableNodeManager->DeepAllocCopy(interp->scopeStack[i + parent_scope_start_index]);
+			}
 
 			if(interp->bottomOfScopeStack)
 				break;

src/Amalgam/interpreter/Interpreter.h

@@ -90,7 +90,8 @@ class Interpreter
 	//but if not, it will attempt to put an appropriate unique associative array on scopeStack
 	//if interpret_with_new_scope will push the new scope before interpreting if needed (e.g., ENT_LET),
 	// otherwise it will push after interpreting (e.g., ENT_CALL)
-	void InterpretAndPushNewScopeStackNode(EvaluableNode *new_context_node, bool interpret_with_new_scope);
+	//if scope_break is true, then it will break the scope from all below it and not give access below
+	void InterpretAndPushNewScopeStackNode(EvaluableNode *new_context_node, bool interpret_with_new_scope, bool scope_break);
 
 	//pops the top context off the stack
 	//if returning_unique_value, then can potentially free the whole scope
@@ -351,6 +352,9 @@ class Interpreter
 	#endif
 	)
 	{
+	#ifdef MULTITHREAD_SUPPORT
+		bool hit_scope_break = false;
+	#endif
 		//find appropriate context for symbol by walking up the stack
 		for(auto it = rbegin(scopeStack); it != rend(scopeStack); ++it)
 		{
@@ -396,11 +400,19 @@ class Interpreter
 				return ScopeStackSymbolLocation{
 					&found->second, mcn, it == rbegin(scopeStack), is_freeable, is_freeable_top_node};
 			}
+
+			if((*it)->IsScopeBreak())
+			{
+			#ifdef MULTITHREAD_SUPPORT
+				hit_scope_break = true;
+			#endif
+				break;
+			}
 		}
 
 	#ifdef MULTITHREAD_SUPPORT
 		//need to search further down the stack if appropriate
-		if(!bottomOfScopeStack && callingInterpreter != nullptr)
+		if(!hit_scope_break && !bottomOfScopeStack && callingInterpreter != nullptr)
 		{
 			bool top_is_next_stack = (scopeStack.size() == 0);
 			auto symbol_location = callingInterpreter->GetScopeStackSymbolLocation(

src/Amalgam/interpreter/OpcodesControlFlow.cpp

@@ -155,7 +155,7 @@ static OpcodeInitializer _ENT_CALL(ENT_CALL, &Interpreter::InterpretNode_ENT_CAL
 		OpcodeDetails::ParameterGroup({"return_warnings", OpcodeDetails::DataType::BOOL, true})
 	};
 	d.returns = OpcodeDetails::DataType::ANY_BASIC;
-	d.description = R"(Evaluates `function` after pushing the `params` assoc onto the scope stack.  If `constraints` is specified and not false or null, it will constrain execution.  If `constraints` is true or an assoc, it will default all constraints to be on at reasonable values for small execution without access to any data beyond `params`.  They optional key-value combinations for `constraints` are as follows.  If "max_node_operations" is specified, it represents the number of operations that are allowed to be performed. If "max_node_operations" is 0, then an infinite of operations will be allotted, up to the limits of the current calling context.  If "max_node_allocations" is specified, it represents the maximum number of nodes that are allowed to be allocated, limiting the total memory, up to the current calling context's limit.   If "max_node_allocations" is 0 and the caller also has no limit, then there is no limit to the number of nodes to be allotted as long as the machine has sufficient memory.  Note that if "max_node_allocations" is specified while in a multithreaded environment, if the collective memory from all the executing threads exceeds the average memory specified by "max_node_allocations", that may trigger a memory limit for the call.  If "max_operation_depth" is 0 or infinite and the caller also has no limit, then there is no limit to the depth that opcodes can execute, otherwise "max_operation_depth" limits how deep nested opcodes will be called. If `return_warnings` is true (default is false), the result will be a tuple of the form [value, warnings, performance_constraint_violation], where warnings is a list of all warnings, and perf_constraint_violation is a string denoting the performance constraint exceeded (or .null if none)).  The keys "read_access" and "write_access" are boolean and control whether the execution can read from or write to entities and access their relevant permissions (e.g., to load files, make system calls).  If the parameter `return_warnings` is false, just the value will be returned.)";
+	d.description = R"(Evaluates `function` after pushing the `params` assoc onto the scope stack.  If `params` is missing or null, then it will allow access to the scope stack of the caller.  If `constraints` is specified and not false or null, it will constrain execution.  If `constraints` is true or an assoc, it will default all constraints to be on at reasonable values for small execution without access to any data beyond `params`.  They optional key-value combinations for `constraints` are as follows.  If "max_node_operations" is specified, it represents the number of operations that are allowed to be performed. If "max_node_operations" is 0, then an infinite of operations will be allotted, up to the limits of the current calling context.  If "max_node_allocations" is specified, it represents the maximum number of nodes that are allowed to be allocated, limiting the total memory, up to the current calling context's limit.   If "max_node_allocations" is 0 and the caller also has no limit, then there is no limit to the number of nodes to be allotted as long as the machine has sufficient memory.  Note that if "max_node_allocations" is specified while in a multithreaded environment, if the collective memory from all the executing threads exceeds the average memory specified by "max_node_allocations", that may trigger a memory limit for the call.  If "max_operation_depth" is 0 or infinite and the caller also has no limit, then there is no limit to the depth that opcodes can execute, otherwise "max_operation_depth" limits how deep nested opcodes will be called. If `return_warnings` is true (default is false), the result will be a tuple of the form [value, warnings, performance_constraint_violation], where warnings is a list of all warnings, and perf_constraint_violation is a string denoting the performance constraint exceeded (or .null if none)).  The keys "read_access" and "write_access" are boolean and control whether the execution can read from or write to entities and access their relevant permissions (e.g., to load files, make system calls).  If the parameter `return_warnings` is false, just the value will be returned.)";
 	d.examples = MakeAmalgamExamples({
 		{R"&((let
 	{
@@ -313,7 +313,10 @@ EvaluableNodeReference Interpreter::InterpretNode_ENT_CALL(EvaluableNode *en, Ev
 		profiling_call = true;
 	}
 
-	InterpretAndPushNewScopeStackNode(ocn.size() > 1 ? ocn[1] : nullptr, false);
+	if(ocn.size() > 1 && !EvaluableNode::IsNull(ocn[1]))
+		InterpretAndPushNewScopeStackNode(ocn.size() > 1 ? ocn[1] : nullptr, false, true);
+	else
+		InterpretAndPushNewScopeStackNode(ocn.size() > 1 ? ocn[1] : nullptr, false, false);
 
 	//call the code
 	auto result = InterpretNode(function, immediate_result);

src/Amalgam/interpreter/OpcodesVariableDefinitionAndModification.cpp

@@ -104,7 +104,7 @@ EvaluableNodeReference Interpreter::InterpretNode_ENT_LET(EvaluableNode *en, Eva
 	if(ocn_size == 0) [[unlikely]]
 		return EvaluableNodeReference::Null();
 
-	InterpretAndPushNewScopeStackNode(ocn[0], true);
+	InterpretAndPushNewScopeStackNode(ocn[0], true, false);
 
 	//run code
 	EvaluableNodeReference result = EvaluableNodeReference::Null();
@@ -1359,8 +1359,6 @@ static OpcodeInitializer _ENT_STACK(ENT_STACK, &Interpreter::InterpretNode_ENT_S
 	)
 	{x 1}
 ))&", R"([
-	{}
-	{x 1}
 	{a 1}
 ])"}
 		});